This post is dedicated to provide detailed guidelines to Area Chairs of CVPR 2009 as to how the process will unfold and what are their responsibilities.  Area Chairs should also see

• The Reviewer Guidelines as they have information about what is expected of reviewers, information about ethics of reviewing and best practices.  
• The AC Instructions (Instructions of how to use the reviewing site)
• The AC FAQ (Some of the recently asked questions and their answers)
• Google Calendar  (XML, ICAL, HTML) that you can subscribe to and see the schedule.

Role of Area Chairs.

The ACs help identify suitable reviewers, assign reviewers to papers, and evaluate the reviews as well as the authors’ responses. For CVPR09, our goal is to provide the highest quality reviews to authors and for this we hope to limit the number of papers assigned per AC to about 30-35 . This will allow each AC to play a more active role in ensuring higher quality and more timely reviews, and in initiating discussions amongst reviewers (anonymously), and in looking at authors’ responses to the reviews via the online rebuttal process. At the AC meeting in February 2009, ACs will work with other ACs (in pairs and then in panels) to decide which papers to accept and write consolidation reports to justify the decisions. Needless to say this is a very important part of the Academic process of Peer Reviewing and has a direct impact on quality of work in Computer Vision. All of the Area Chairs for CVPR 2009 are committed to doing this task in a timely, professional and ethical manner.

Best Practices of BEING an AC

We just want to remind all ACs of some of the basic (and very obvious) practices of being an AC.  

• You have a strong influence on a decision on a paper.  Take it seriously and be fair.
• DO NOT talk to any other AC about papers assigned to you until you are told about who is on your panel and as buddy.  There maybe many people who are AC who maybe conflicted with the paper you are an Area Chair for, and best if they don’t know that.  We will provide some details on how to do this as we get closer to the AC meeting. 
• The Program Co-Chairs did not submit any papers, so they are not in direct conflict with any papers and they know all authors of all papers and also all reviewers/AC. However, they will NOT be involved with any decisions of paper from institutions they are in conflict with. So you can always send them a note about any paper and if one of them is conflicted, another one will respond to you without including the conflicted PC Chair. 
• DO NOT talk to any other AC about your own paper (the paper you are an author on) or a paper you have some conflict with, during this whole process, unless that AC and you are already collaborators and both of you have a already defined conflict.  Do not lobby for your paper, hand out copies to other AC and show results or discuss your paper.  
• Advocate for papers your like, DO not kill papers you don’t like, unless of course you see a technical detail that is wrong and that would be un-acceptable to accept. Convince others to see your point of view.
• Be professional and willing to listen to other reviewers and ACs. DO not give in to undue influence from anyone.
• Do remember, (we assume with just cause, knowing the high quality reviewers we have) that the reviewers have spend time reading the paper. It is not fair to dismiss any review without looking at other reviews and reading the paper yourself to come of a evaluation. Outright rejecting a review is not fair.  Please work hard to make sure that if you are over ruling a review of someone that you have some serious grounds to do so (just saying “this reviewer is plain wrong” without justifying is not something we want!). 
• Senior members of the committee, guide and advise the younger and first-time members.  First-time members, please feel free to ask for help as needed. 

Make Sure You Assigned the Papers Appropriate for YOU.

The Program Chairs are going to assign papers to you based on (a) papers from authors not in conflict with you (see conflict of interest in Reviewer Guidelines), (b) subject areas you chose when you registered on the review site, and (c) subject areas, the authors chose, when they registered the paper. As can be imagined, this will NOT be perfect as primarily as some authors have differing takes on some subject areas then we do and also, sometime the Program Chairs have to do some load-balancing.  

Area Chairs should look through all the papers assigned to them and ensure that they are (1) not conflicted and (2) are knowledgeable in the subject area. Please look at the PDF of the papers in addition to the title/abstract of the paper to ensure this.  You should look at these papers to identify reviewers too.

Assigning Reviewers.

Remember, we want you to recommend 5 reviewer for each paper assigned to you. We will try to find 3 of these for the paper, trying to balance reviewer load and your choices. Here are some suggestions on how to proceed.  Remember, spend good time doing this, as it will really help with your role as an AC.  Dedicate parts of day to do this.  It can take anywhere from 4-6 minutes a paper do this, if you look at the paper and other info to help make a decision. Considering a max load of 35 papers, you can see, that this may take some time.  Trust us, it will be worth it. See Review Instructions for details of how to use the system to assign reviewers. 

• You have been chosen as an Area Chair because of your specific expertise in certain area of Vision and also you general expertise in Computer Vision. When you look at a paper, you may just come up with a name or three of ideal reviewers.  Consider these people as the reviewers for this, but do also consider things like, (a) are they current in the field and (b) will they do a good job in a timely manner.
• Look at the paper, especially the Introduction, Related Work, and Citations at the end.  See who they refer to and whose work they are building on.  On the cited people would be an ideal reviewers
• Do remember, that the system will give you some recommendation, based on simple heuristic by matching subject areas. This will provide a good starting point, but will not be perfect as some folks are not that good at choosing keywords (authors and reviewers).   
• Usually, we suggest that you choose a reviewer you know and trust. But vision community us large (how else would we get 1450+ submissions). If the system recommends someone you do not know, look for their webpage or look for their papers on on ACM or IEEE digital libraries or Google Scholar and the like.
• A new feature has been added to the system to allow you to add a new reviewer to a paper.  New reviewer, as an a reviewer that we did already get registered into the system before the deadline. Please use this carefully as we do not want to have a lots of outstanding review as reviewers were added in late in the process. See Review Instructions for info on how to add a reviewer on the fly.
• Again, spend good time doing this.

Keeping Track of Reviews and Following up with Reviewers.

After you have ranked reviewers, the Program Chairs, with the review system’s help will distribute papers to the reviewers. During this phase, we recommend that you log into the system to check if any reviews are being entered.  Do remember that the reviewers DO NOT KNOW your identity, so best if you do not contact them and/or identify yourself.  Yet another form “anonymity” in the review process, as we don’t want the reviewers to be unduly influenced by anyone.  If reviews are showing up, look at them and see if (a) the reviewer has added sufficient detail and (b) has appropriate tone in the review, and most importantly (e) providing a knowledgeable review. Contact the reviewers via the review system to get them to clarify issues from the review.  Doing this during the process may prevent you from doing it all at the deadline of when reviews are due.

After the review deadline has passed, check if all reviews are in.  If not, send them a note and nicely DEMAND that they finish the reviews soon.  The system will also send reminders to them.

Reaching Consensus

Coming Soon

How to best use the Rebuttal Process?

Coming Soon

Working in AC Pairs (”buddies”).

Coming Soon

Working in AC Panels

Dos and Don’ts at the AC Meeeting

Making Decisions

Consolidation Reports

Area Chair Consolidation Reports are the MOST CRUCIAL aspect of the review process of CVPR.  This is where the Area Chair justifies his/her recommendation to accept/reject a paper. The Program Chairs will read all Consolidation Reports and we have committed ourselves to ensure that these reports are representative of a good quality review process, which we are guaranteeing to the Vision community resting on the Area Chair’s shoulders.  These reports should highlight why the decision was reached. If all reviewers agree on a paper, then this consolidation report can be simple, but feel free to encourage authors and provide constructive feedback.  If there is even a slight disagreement on the reviews, it is your job to clarify why how the disagreement was resolved. Again, just saying “reviewer 1 was wrong” without any reasons is not acceptable.  Explain why.  For the most divergent reviews, explain the process you went through to reach a decision and if you and others read the paper, try to add some details from your and others reviews with the consolidation reports.  Over ruling all three reviewers to make a decision on a paper is not acceptable to us.  If you strongly feel that the paper deserves that make sure to discuss it in detail with your AC Buddy and also in your AC Panel and try to solicit an additional review.  In such cases, (a) you should read the paper completely and (b) provide a detailed consolidation report.

Tags: Reviewing

This entry was posted on Saturday, November 1st, 2008 at 12:48 pm and is filed under Guidelines, Reviewing. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.